We study the problem of online adaptive policy selection for nonlinear time-varying discrete-time dynamical systems.

Prior work on OCO with memory is typically limited in one of two ways.

We study a challenging form of Smoothed Online Convex Optimization, a.k.a.

We analyze Decentralized Online Optimization algorithms using the Performance Estimation Problem approach which allows, to automatically compute exact worst-case performance of optimization algorithms. Our analysis shows that several available performance guarantees are very conservative, sometimes by multiple orders of magnitude, and can lead to misguided choices of algorithm. Moreover, at least in terms of worst-case performance, some algorithms appear not to benefit from inter-agent communications for a significant period of time. We show how to improve classical methods by tuning their step-sizes, and find that we can save up to 20% on their actual worst-case performance regret.

It aims to solve a large-scale optimization problem by decomposing it into smaller, more tractable subproblems that can be solved iteratively and in parallel by a network of interconnected agents through communication. Most traditional works on distributed optimization focus on static problems, making them unsuitable for dynamic tasks arising in real-world applications, such as networked autonomous vehicles, smart grids, and online machine learning, among others [8]. Online optimization, which addresses time-varying cost functions, plays a vital role in solving dynamic problems in timely application fields [58, 29, 21, 3]. In many practical scenarios, such as machine learning with information streams [46], the objective functions of optimization problems change over time, making them inherently dynamic [49, 58]. Online learning has emerged as a powerful method for handling sequential decision-making tasks in dynamic contexts, enabling real-time operation while ensuring bounded performance loss in terms of regret [12]. Regret is the gap between the cumulative objective value achieved by the online algorithm and that of the optimal offline solution [19, 44]. In the literature, two types of regret are commonly considered, i.e., static and dynamic regret.

Prior work on OCO with memory is typically limited in one of two ways.

We consider the problem of online optimization, where a learner chooses a decision from a given decision set and suffers some loss associated with the decision and the state of the environment. The learner's objective is to minimize its cumulative regret against the best fixed decision in hindsight. Over the past few decades numerous variants have been considered, with many algorithms designed to achieve sub-linear regret in the worst case. However, this level of robustness comes at a cost. Proposed algorithms are often over-conservative, failing to adapt to the actual complexity of the loss sequence which is often far from the worst case. In this paper we introduce a general algorithm that, provided with a "safe" learning algorithm and an opportunistic "benchmark", can effectively combine good worst-case guarantees with much improved performance on "easy" data. We derive general theoretical bounds on the regret of the proposed algorithm and discuss its implementation in a wide range of applications, notably in the problem of learning with shifting experts (a recent COLT open problem). Finally, we provide numerical simulations in the setting of prediction with expert advice with comparisons to the state of the art.

This paper proposes a novel gradient estimator that performs a weighted average (similar to momentum) of past and new gradients to estimate the gradient at the current iterate. To motivate their estimator, the authors demonstrate that the SG method with momentum does not decrease the variance unless the momentum parameter is increased like 1 – 1/t. The IGT estimator is then derived by considering the quadratic case (where the Hessian matrix is fixed for all individual functions) with the goal of estimating the "true" online gradient (the simple average over all previously seen gradients). In order to compensate for the bias of past gradients, the new gradient is notably evaluated at an extrapolated point, not at the current point. This derived estimator yields an O(1/t) reduction in variance, yielding a theoretical result that may be interpreted as linear convergence to a neighborhood that shrinks as O(1/t) with constant steplength for quadratic problems.